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Program Description^ 



University of Chicago Schooi Mathematics Project (UCSMP) Aigebra 
is a one-year course covering three primary topics: (1) iinear and 
quadratic expressions, sentences, and functions; (2) exponentiai 
expressions and functions; and (3) iinear systems. Topics from 
geometry, probabiiity, and statistics are integrated with the 



appropriate aigebra. Probiem-solving and real-worid applications 
are used throughout to deveiop and maintain basic skilis and con- 
cepts. Computer aigebra system (CAS) technology is used in the 
classroom to aid in the deveiopment of properties and skilis, and 
graphing calculators are used to complete assignments at home. 



Research One study of UCSMP Aigebra meets the What Works Clearinghouse Based on this one study, the WWC considers the extent of evi- 
(WWC) evidence standards with reservations. The study includes 36 dense for UCSMP Aigebra to be small for the math achievement 
eighth-grade students in a Nebraska junior high school. This study domain, 
used the first edition of UCSMP Aigebra as the intervention.^ 

Effectiveness UCSMP Algebra was found to have no discernible effects on math achievement. 



Math achievement 
Rating of effectiveness No discernible effects 
Improvement index'^ Average: -6 percentile points 

1. This report has been updated from the previous version (posted March 26, 2007) to include reviews of 10 studies that have been released since 2005. All 
10 studies are not within the scope of the protocol. A complete list and disposition of all studies reviewed is provided in the references. Additionally, one 
study that met standards with reservations in the previous version (Thompson, Senk, Witonsky, Usiskin, & Kaeley, 2006) will now be eligible for review 
as part of the WWC high school math topic area. (The protocol for the middle school math topic area was revised to narrow the scope from examining 
any students in grades 6 to 9 to examining only those students who are attending middle schools or junior high schools. Studies examining students in 
grade 9 who are attending high school are included in the high school math topic area.) 

2. The descriptive information for this program was obtained from a publicly available source: the program’s website (https://www.wrightgroup.com/ 
ucsmp/index.html?PHPSESSID=50e67838d5e498ac7fed20fc7acf10ea&gid=228, downloaded August 2008). The WWC requests developers to review 
the program description sections for accuracy from their perspective. Further verification of the accuracy of the descriptive information for this program 
is beyond the scope of this review. 

3. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

4. These numbers show the average student-level improvement indices for all findings across the study. 
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Additional program Developer and contact 

information Developed by the University of Chicago School Mathematics 
Project, UCSMP Algebra is distributed by the Wright Group/ 
McGraw-Hill. Address: 220 East Danieldale Road, DeSoto, 

TX 75115. Email: WrightGroup@McGraw-Hill.com. Web: 
https://www.wrightgroup.com/ucsmp/index.html?PHPSESSID=5 
0e67838d5e498ac7fed20fc7acf10ea&gid=228. Telephone: 

(800) 523-2371. 

Scope of use 

The first edition of UCSMP Algebra was developed and tested 
between 1985 and 1988, and the second edition was developed 
and tested between 1992 and 1994. The third edition was 
developed and tested between 2005 and 2007 and is now avail- 
able through the Wright Group/McGraw-Hill. According to the 
developers at the University of Chicago, 3.5 to 4 million students 
in elementary, middle, and high schools are currently using 
UCSMP materials and curricula. The number of students using 
only UCSMP Algebra is not available. 



Teaching 

The UCSMP Algebra course includes the student book, teacher’s 
edition book and teacher resources, assessment resources, and 
technology resources. Almost all lessons in the student book con- 
tain activities, full examples, guided examples, and “quiz yourself” 
questions to create a more active classroom. Questions in each 
lesson cover the lesson concepts, extensions and applications of 
those concepts, and previous lessons. The teacher’s edition book 
contains additional examples as well as suggestions for differentia- 
tion to accommodate the broad population of students in algebra 
courses. Computer algebra system (CAS) technology is used in the 
classroom to aid in the development of properties and skills, and 
graphing calculators are used to complete assignments at home. 

Cost 

A student textbook costs $63.00. A bundled, complete teacher 
resource package consisting of the Teacher’s Edition (Volumes 
1 & 2), Teacher’s Resources (Volumes 1 & 2), Assessment CD 
Rom, and “eTe with Answers and Solutions” (Volumes 1 & 2) 
costs $346.50. See the publisher’s website for pricing of individual 
teacher resource items. 



Research Thirteen studies reviewed by the WWC investigated the effects 
of UCSMP Algebra. One study (Peters, 1992) is a randomized 
controlled trial with randomization problems that meets WWC 
evidence standards with reservations. The remaining 12 studies 
do not meet either WWC evidence standards or eligibility screens. 

Meets evidence standards with reservations 

Peters (1992) conducted a randomized controlled trial design 
with randomization problems.® The study’s sample included 
36 “math-talented” students from one junior high school in 



Nebraska.® Most of the students were Caucasian. The district 
borders two large cities (Lincoln and Omaha) and has a mix of 
students living in rural and suburban locations. Students in the 
intervention group used the UCSMP Algebra first edition text- 
book, while students in the comparison group used the Saxon 
Middle School Math curriculum. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the What Works Clearinghouse 



5. Peters (1992) compared UCSMP Algebra with Saxon Middle School Math. The author indicates that a random selection of numbers was used to divide 
participants between the intervention and comparison groups. However, the assignment of students was altered to accommodate scheduling difficulties 
and student requests for other course offerings. Therefore, the study meets standards with reservations, according to WWC criteria. 

6. The “math-talented” designation is based on teacher recommendations and prior academic achievement. No information is provided on the specific 
thresholds that were used in delineating the math-talented criteria; however, all students in the sample scored at or above the 87th percentile on the 
California Achievement Test total math battery. 
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RGSCarch (continued) Extent of Evidence Categorization Scheme). The extent of The WWC considers the extent of evidence for UCSMP 

evidence takes into account the number of studies and the Algebra to be small for math achievement, 

totai sample size across the studies that meet WWC evidence 
standards with or without reservations.^ 



Effectiveness 



Findings 

The WWC review of middle school math curricula addresses 
student outcomes in the math achievement domain. The findings 
below present the author’s estimates and WWC-calculated 
estimates of the size and the statistical significance of the effects 
of UCSMP Algebra on students.® 

Peters (1992) reported no statisticaiiy significant differences 
in Orieans-Hanna Algebra Prognosis test scores between the 
UCSMP Algebra group and the Saxon Middle School Math 
group. Further, the effect size was neither statisticaiiy significant 
nor large enough to be considered substantively important by 
WWC criteria (at least 0.25). 



In sum, one study of UCSMP Algebra, first edition, showed an 
indeterminate effect. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentiaily positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statisticai significance of the findings (as calcuiated by 
the WWC), the size of the difference between participants in the 
intervention and the comparison conditions, and the consistency in 
findings across studies (see the WWC Intervention Rating Scheme). 



The WWC found UCSMP 
Algebra to have no 
discernible effects on math 
achievement. 



Improvement index 

The WWC computes an improvement index for each individuai 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technicai 
Detaiis of WWC-Conducted Computations). The improvement 
index represents the difference between the percentiie rank 
of the average student in the intervention condition versus 
the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is entireiy based on the size of the effect, regardiess of 
the statisticai significance of the effect, the study design, or the 
analysis. The improvement index can take on values between 



-50 and +50, with positive numbers denoting resuits favorable to 
the intervention group. 

The average improvement index for math achievement is -6 
percentiie points in the study. 

Summary 

The WWC reviewed 13 studies on UCSMP Algebra. One of 
these studies meets WWC evidence standards with reservations; 
the remaining 12 studies do not meet either WWC evidence 
standards or eligibiiity screens. Based on this one study, the 
WWC found no discernible effects on students’ math achievement. 
The conclusions presented in this report may change as new 
research emerges. 



7. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the Intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined for UCSMP Algebra is in Appendix A5. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate 
the statistical significance, see Technical Details of WWC-Conducted Computations. In the case of Peters (1992), no correction for clustering or multiple 
comparisons was needed. 
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For more information about specific studies and WWC calculations, please see the WWC UCSMP Algebra Technical Appendices. 
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Appendix 



Appendix A1 Study Characteristics: Peters, 1992 (randomized controiied triai with randomization prohiems) 



Characteristic 


Description 


Study citation 


Peters, K. G. (1992). Skill performance comparability of two algebra programs on an eighth-grade population. Dissertation Abstracts International, 54(01), 77A. (UMI No. 9314428) 


Participants 


The study included 36 students in grade 8 in two classrooms in one school during the 1991/92 school year. All of the students were “math-talented” based on teacher 
recommendations and prior academic achievement. No information is provided on the specific thresholds that were used in delineating the math-talented criteria; however, 
all students in the sample scored at or above the 87th percentile on the California Achievement Test total math battery. The sample consisted of 20 girls (9 treatment, 11 
comparison) and 16 boys (8 treatment, 8 comparison). 


Setting 


The study took place in one junior high school in Nebraska. The district borders two large cities (Lincoln and Omaha) and has a mix of students living in rural and suburban 
locations. Students were randomly assigned to one of two classrooms (one intervention classroom and one comparison classroom).^ The same teacher taught both the 
intervention and comparison groups. 


Intervention^ 


Participants in the intervention group were taught the UCSMP Algebra curriculum during the 1991/92 school year. The intervention curriculum was monitored weekly by the 
researcher to help maintain fidelity of implementation. 


Comparison 


Participants in the comparison group were taught using the Saxon Middle School Math curriculum for eighth-grade students (Algebra 1/2). Students in this group participated 
in daily sessions for one academic year. In each session, the teacher introduced a new concept incrementally, and students had opportunities to practice the new concept and 
past concepts during each session. Students were assessed every fifth lesson. The Saxon Math curriculum is designed to cover 120 lessons in one year. 


Primary outcomes 
and measurement 


The primary outcome measure is the Orleans-Hanna Algebra Prognosis Test.^ The pretest administration occurred in August 1991, and the posttest administration occurred 
in May 1992. For a more detailed description of this outcome measure, see Appendix A2. 


Staff/teacher training 


The study noted that the teacher who taught both study groups did not have prior experience with the intervention or comparison curricula but had read extensively about 
both teaching formats. The teacher participated in a one-week summer workshop on UCSMP Algebra, and in two additional one-day workshops given by local consultants 
on the curricula used in this study. Further, agreed-upon components of both the intervention and comparison curricula were monitored on a weekly basis by the researcher 
to help maintain the integrity of implementation. 



1 . The author indicates that a random seiection of numbers was used to divide participants between the intervention and comparison groups. However, the assignment of students was aitered to 
accommodate scheduiing difficuities and student requests for other course offerings. The anaiysis sample includes 17 students in the UCSMP Algebra group and 19 in the Saxon Math group. 
The study author demonstrated the baseline equivalence of the UCSMP Algebra and Saxon Math groups at pretest. 

2. The same teacher taught both the intervention and comparison groups. Because both the intervention and comparison curricula were monitored on a weekly basis by the researcher to help 
maintain the integrity of implementation, and because there is no indication in the study to assume that the teacher was biased toward one of the conditions, this design was accepted for review. 

3. The author described only the Orleans-Hanna Algebra Prognosis Test as the measure of student math achievement. The study also examined four study-generated criterion unit tests, not from 
the Orleans-Hanna Algebra Prognosis Test, designed to descriptively measure student understanding of algebraic components. However the author did not provide information on the reliability 
or validity of these four tests. Accordingly, analyses based on these four unit tests were not considered in this version of the report. 
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Appendix A2 Outcome measure for the math achievement domain 



Outcome measure 


Description 


Orleans-Hanna Algebra 
Prognosis Test 


This nationally normed test consists ot 60 multiple-choice items based on nine model lessons and five questionnaire items that require students to report their course grades 
and predict their final grade if they were to take aigebra. In contrast to an achievement test, students are required to answer questions by following a procedure or set of 
operations using mathematical or verbal expressions parallel to but different from those contained in the model lessons. This test is often used to predict the ability to succeed 
in a first-year algebra course (as cited in Peters, 1992). For Peters (1992), pretest scores on the Orleans-Flanna Prognosis Test were trom an August 1991 administration and 
posttest scores were from a May 1992 administration. 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
students) 


UCSMP Algebra Comparison 

group group 


Mean 

difference^ 
{UCSMP Algebra- 
comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 



Peters, 1992 (randomized controlled trial with randomization problems)^ 








Orleans-Hanna Algebra Grades 2/36 95.02® 95.63® -0.61 

Prognosis Test (math-talented) (4.09) (4.53) 


-0.14 


ns 


-6 


Average for math achievement (Peters, 1992)'° 


-0.14 


ns 


-6 


Oomain average for math achievement across all studies'” 


-0.14 


ns 


-6 



ns = not statistically significant 



1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices for the math achievement domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 

had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 

The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study author or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple com- 
parisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate statistical significance, see Technical Details of 
WWC-Conducted Computations. In the case of Peters (1992), no correction for clustering or multiple comparisons was needed. 

8. The intervention group value from Peters (1992) is the comparison group mean plus the difference in mean gains between the intervention (UCSMP Algebra) and control groups. 

9. The comparison group mean from Peters (1992) is unadjusted. 

10. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 
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Appendix A4 UCSMP Algebra rating for the math achievement domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of math achievement, the WWC rated UCSMP Algebra as having no discernibie effects. 



Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statisticaiiy significant or substantively important effect, either positive or negative. 

Met. No studies showed statisticaiiy significant or substantively important positive or negative effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies showed statisticaiiy significant positive effects. 

AND 

• Criterion 2: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. No studies showed statisticaiiy significant or substantively important negative effects. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important positive effect. 

Not met. No studies showed a statistically significant or substantiveiy important positive effect. 

AND 

• Criterion 2: No studies showing a statisticaiiy significant or substantiveiy important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statisticaiiy significant or substantively important positive effects. 

Not met. The study that evaluated math achievement and met WWC standards with reservations showed indeterminate effects. 

Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following criteria. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantiveiy important negative effect, but no more such studies than the number showing a statisticaiiy significant or substantiveiy important positive effect. 

Not met. No studies showed a statisticaiiy significant or substantiveiy important effect, either positive or negative. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. No studies showed statistically significant or substantively important effects, either positive or negative. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. 



WWC Intervention Report University of Chicago School Mathematics Project (UCSMP) Algebra 



March 2009 



8 











Appendix A5 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schools 


Sample size 


Students 


Extent of evidence^ 


Math achievement 


1 


1 




36 


Small 



1 . A rating of “medium to iarge” requires at ieast two studies and two schoois across studies in one domain, and a totai sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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